Papers with model inference speed
SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for quantization of models are too complicated and can cause performance damage. |
| Approach: | They propose a self-adaptive mixed-precision (SAMP) toolkit to automatically control quantization rate by a mixed-presence architecture to balance model accuracy and efficiency. |
| Outcome: | The proposed toolkit has a higher speedup than PyTorch and FasterTransformer while ensuring the required accuracy. |
PuMer: Pruning and Merging Tokens for Efficient Vision Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale vision language models use Transformers to perform cross-modal interactions . state-of-the-art models are memory intensive and expensive due to quadratic complexity . |
| Approach: | They propose a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text. |
| Outcome: | The proposed framework improves inference speed and memory footprint on four vision language tasks. |